Assisted perception for autonomous vehicles

ABSTRACT

Disclosed herein are systems and methods for providing supplemental identification abilities to an autonomous vehicle system. The sensor unit of the vehicle may be configured to receive data indicating an environment of the vehicle, while the control system may be configured to operate the vehicle. The vehicle may also include a processing unit configured to analyze the data indicating the environment to determine at least one object having a detection confidence below a threshold. Based on the at least one object having a detection confidence below a threshold, the processor may communicate at least a subset of the data indicating the environment for further processing. The vehicle is also configured to receive an indication of an object confirmation of the subset of the data. Based on the object confirmation of the subset of the data, the processor may alter the control of the vehicle by the control system.

CROSS REFERENCE TO RELATED APPLICATION

The present application claims priority to U.S. Patent Application Ser.No. 61/909,887, filed on Nov. 27, 2013, the entire contents of which areherein incorporated by reference.

BACKGROUND

Unless otherwise indicated herein, the materials described in thissection are not prior art to the claims in this application and are notadmitted to be prior art by inclusion in this section.

A vehicle could be any wheeled, powered vehicle and may include a car,truck, motorcycle, bus, etc. Vehicles can be utilized for various taskssuch as transportation of people and goods, as well as many other uses.

Some vehicles may be partially or fully autonomous. For instance, when avehicle is in an autonomous mode, some or all of the driving aspects ofvehicle operation can be handled by a vehicle control system. In suchcases, computing devices located onboard and/or in a server networkcould be operable to carry out functions such as planning a drivingroute, sensing aspects of the vehicle, sensing the environment of thevehicle, and controlling drive components such as steering, throttle,and brake. Thus, autonomous vehicles may reduce or eliminate the needfor human interaction in various aspects of vehicle operation.

SUMMARY

The present disclosure provides methods and apparatuses that enable asystem to take advantage of human operators (or a more powerful computeroperators) as part of the object identification of the autonomousvehicle. During the operation of an autonomous vehicle, the vehicle mayidentify various objects (e.g., features) it detects. However, theautonomous vehicle may have a hard time detecting some objects. Thus,the methods and apparatuses presented herein enable further processingof objects by using a human operator or a more powerful computer system.

An apparatus, such as an autonomous vehicle, disclosed herein includesboth a sensor unit and a control system. The sensor unit of theapparatus may be configured to receive data representing objects of anenvironment of the apparatus. The control system of the apparatus isconfigured to operate the apparatus. The apparatus also includes one ormore processors. The one or more processors may be configured todetermine, based on the received data, at least one object of theenvironment that has a detection confidence below a threshold. Thedetection confidence may indicate a likelihood that the determinedobject is correctly identified in the environment. Based on the at leastone object having a detection confidence below the threshold, the atleast one processor may communicate, from the received data, image dataof the at least one object for further processing to a remote operator.Additionally, based on the at least one object having a detectionconfidence below the threshold, the at least one processor may determineat least one natural-language question associated with the at least oneobject having a detection confidence below the threshold. The at leastone processor may also communicate the at least one natural-languagequestion to the remote operator. The at least one processor may alsoreceive an answer to the at least one natural-language question from theremote operator providing an object indication, and provide instructionsto control the apparatus by the control system based on the objectindication.

A method disclosed herein includes receiving data by a sensor, where thedata represents objects of an environment of an autonomous vehicle. Themethod also includes operating the autonomous vehicle by a controlsystem. Additionally, the method includes determining, by a processor,based on the received data, at least one object of the environment thathas a detection confidence below a threshold, where the detectionconfidence indicates a likelihood that the determined object iscorrectly identified in the environment. Based on the at least oneobject having a detection confidence below the threshold, the methodincludes the processor communicating, from the received data, image dataof the at least one object for further processing to a remote operator.Additionally, based on the at least one object having a detectionconfidence below the threshold, the method includes determining at leastone natural-language question associated with the at least one objecthaving a detection confidence below the threshold. The method alsoincludes communicating the at least one natural-language question to theremote operator. The method also includes the processor receiving ananswer to the at least one natural-language question from the remoteoperator providing an object indication. Further, as a part of themethod, the processor may provide instructions to control the autonomousvehicle by the control system based on the object indication.

Also disclosed herein is an article of manufacture including anon-transitory computer-readable medium having stored thereon programinstructions that, when executed by a processor in a vehicle system,causes the vehicle system to perform operations. The operations includereceiving data by a sensor, where the data represents objects of anenvironment of an autonomous vehicle. Additionally, the operationsinclude determining, based on the received data, at least one object ofthe environment that has a detection confidence below a threshold, wherethe detection confidence indicates a likelihood that the determinedobject is correctly identified in the environment. Based on the at leastone object having a detection confidence below the threshold, theoperations include communicating, from the received data, image data ofthe at least one object for further processing to a remote operator.Additionally, based on the at least one object having a detectionconfidence below the threshold, the operations include determining atleast one natural-language question associated with the at least oneobject having a detection confidence below the threshold. The operationsinclude communicating the at least one natural-language question to theremote operator. The operations also include receiving an answer to theat least one natural-language question from the remote operatorproviding an object indication. Further, as a part of the operations,the processor may provide instructions to control the autonomous vehicleby the control system based on the object indication.

An apparatus disclosed herein includes a means for receiving data by asensor, where the data represents objects of an environment of anapparatus. The apparatus also include means for operating the apparatusby a control system. Additionally, the apparatus includes means fordetermining, based on the received data, at least one object of theenvironment that has a detection confidence below a threshold, where thedetection confidence indicates a likelihood that the determined objectis present or correctly identified in the environment. Based on the atleast one object having a detection confidence below the threshold, theapparatus includes means for communicating at least a subset of thereceived data for further processing to a remote operator. The apparatusalso include means for receiving an object confirmation from the remoteoperator. Further, the apparatus may also include means for altering theinstructions to control of the apparatus by the control system based onthe object confirmation of the subset of the data.

The foregoing summary is illustrative only and is not intended to be inany way limiting. In addition to the illustrative aspects, embodiments,and objects described above, further aspects, embodiments, and objectswill become apparent by reference to the figures and the followingdetailed description and the accompanying figures.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a functional block diagram illustrating a vehicle, accordingto an example embodiment.

FIG. 2 shows a vehicle, according to an example embodiment.

FIG. 3 shows a method, according to an example embodiment.

FIG. 4A illustrates a top view of an autonomous vehicle operatingscenario, according to an example embodiment.

FIG. 4B illustrates a sensor data representation of the scenario fromFIG. 4A, according to an example embodiment.

FIG. 4C illustrates a video feed taken from the vehicle in FIG. 4A,according to an example embodiment.

FIG. 4D illustrates a graphical interface containing the sensor datarepresentation from FIG. 4B and the video feed from FIG. 4C, accordingto an example embodiment.

FIG. 4E illustrates the graphical interface from FIG. 4D including acontrol menu, according to an example embodiment.

FIG. 5 is a top view of an autonomous vehicle during operation,according to an example embodiment.

FIG. 6 is a schematic diagram of a computer program, according to anexample embodiment.

DETAILED DESCRIPTION

Example methods and systems are described herein. Any example embodimentor feature described herein is not necessarily to be construed aspreferred or advantageous over other embodiments or features. Theexample embodiments described herein are not meant to be limiting. Itwill be readily understood that certain aspects of the disclosed systemsand methods can be arranged and combined in a wide variety of differentconfigurations, all of which are contemplated herein.

Furthermore, the particular arrangements shown in the Figures should notbe viewed as limiting. It should be understood that other embodimentsmight include more or less of each element shown in a given Figure.Further, some of the illustrated elements may be combined or omitted.Yet further, an example embodiment may include elements that are notillustrated in the Figures.

The present disclosure provides methods and apparatuses that enable asystem to incorporate an input from a human operator (or a more powerfulcomputer operator) as part of the object identification of theautonomous vehicle. Specifically, rather than an autonomous vehicledetecting and identifying every object and driving situation that may beencountered, the present approach allows the vehicle to query a humanoperator or a more powerful computing system to help make optimaldecisions. For example, when an object was identified with a lowconfidence by the computer system of the vehicle, the autonomous vehiclemay display an image to a passenger of the vehicle and/or a remote humanoperator to identify the object. The object's identification (e.g.,identification) may then be used by the control system of the autonomousvehicle. In another example, the image may be sent to a remote serverfor identification. Because of the relatively limited computing power onthe vehicle itself, more complex identification may be performed by aremote server and communicated back to the vehicle.

When developing detectors and classifiers for autonomous systems, thereis often a trade-off that can be made between the precision of thesystem (how often it reports it sees an object A when it fact A doesn'texist in reality) and the recall of the system (how often it misses anobject A when the object does exist in reality). A system of the presentdisclosure includes a system bias to have high recall (at the expense oflower precision) so that the system will generally default towardindicating the presence of an object even when the system has lowconfidence. When the system detects an object but is not highlyconfident in the detection of the object, the system can ask a humanoperator (or a more powerful computer) to confirm that the object doesin fact exist. This allows the system to make the right decisions withminimal human input. Thus, the system will generally operate as a highlyaccurate autonomous system, but the system may request human input orother input when the autonomous system has low confidence in anidentification. Additionally, when the system detects an object, but hasa low confidence in the detection, the system may operate as if theobject really exists. A human operator (or other computing system) mayprovide an indication that the detection was incorrect, at which pointthe system will operate as if the erroneous detection was never made.This allows the vehicle to operate with a high precision and affords asafe autonomous operation.

An example of an unusual driving scenario is as follows. There may be anobstacle on the side of a narrow two-lane road, blocking one of the twolanes. The obstacle may not part of a normal identification that thevehicle may perform during most typical operation. Further, the objectmay be trying to direct traffic in an atypical manner (e.g., so thatoncoming traffic and outgoing traffic share the one open lane). If anautonomous vehicle comes up to this scenario, it may detect theexistence of the object, but the autonomous vehicle may have a lowconfidence in detecting the object. For example, the object may be aperson located near the side of the road, the autonomous vehicle mayattempt to identify whether the person is just someone walking in theroad or if the person is attempting to direct traffic and whether he orshe is signaling the autonomous vehicle to drive or to stop. In such acase, the autonomous vehicle could detect that this is likely an unusualevent and send a camera image (or several, and/or other sensor feeds) toa human operator that could quickly look at the situation and confirmthat the autonomous vehicle should stop and wait until it is their turn.The vehicle may also initially operate as if the detected object isindicating the vehicle should stop, while awaiting a confirmation fromthe human operator. This helps ensure a safe operation of the vehicle ina situation where the confidence of the detection is low.

In other embodiments, the sensors used with the identification mayinclude those other than optical cameras. For example, the sensors mayinclude radar, laser, and audio sensors. Information from any of thesesensors may be used to augment the autonomous identification based onhuman input. In one additional example, a microphone on an autonomousvehicle may capture a sound. The identification system may not be ableto detect if the sound is either (i) actually a siren or (ii) part of asong. The present disclosure may use either human interaction or furthercomputing processing by a more powerful remote computer in order todetermine if the sound includes a siren signal.

Example systems within the scope of the present disclosure will now bedescribed in greater detail. An example system may be implemented in ormay take the form of an automobile. However, an example system may alsobe implemented in or take the form of other vehicles, such as cars,trucks, motorcycles, buses, boats, airplanes, helicopters, lawn mowers,earth movers, boats, snowmobiles, aircraft, recreational vehicles,amusement park vehicles, farm equipment, construction equipment, trams,golf carts, trains, and trolleys. Other vehicles are possible as well.

FIG. 1 is a functional block diagram illustrating a vehicle 100,according to an example embodiment. The vehicle 100 could be configuredto operate fully or partially in an autonomous mode. While in autonomousmode, the vehicle 100 may be configured to operate without humaninteraction. For example, a computer system could control the vehicle100 while in the autonomous mode, and may be operable to operate thevehicle an autonomous mode. As part of operating in the autonomous mode,the vehicle may identify objects of the environment around the vehicle.If one or more of the identified objects has an associated confidencebelow a confidence threshold, the computer system may transmit datarelated to the object for further processing. The computer system mayreceive an indication of the object, from either a remote computer or ahuman operator. In response, the computer system may alter the controlof the autonomous vehicle.

The vehicle 100 could include various subsystems such as a propulsionsystem 102, a sensor system 104, a control system 106, one or moreperipherals 108, as well as a power supply 110, a computer system 112, adata storage 114, and a user interface 116. The vehicle 100 may includemore or fewer subsystems and each subsystem could include multipleelements. Further, each of the subsystems and elements of vehicle 100could be interconnected. Thus, one or more of the described functions ofthe vehicle 100 may be divided up into additional functional or physicalcomponents, or combined into fewer functional or physical components. Insome further examples, additional functional and/or physical componentsmay be added to the examples illustrated by FIG. 1.

The propulsion system 102 may include components operable to providepowered motion for the vehicle 100. Depending upon the embodiment, thepropulsion system 102 could include an engine/motor 118, an energysource 119, a transmission 120, and wheels/tires 121. The engine/motor118 could be any combination of an internal combustion engine, anelectric motor, steam engine, Stirling engine. Other motors and/orengines are possible. In some embodiments, the engine/motor 118 may beconfigured to convert energy source 119 into mechanical energy. In someembodiments, the propulsion system 102 could include multiple types ofengines and/or motors. For instance, a gas-electric hybrid car couldinclude a gasoline engine and an electric motor. Other examples arepossible.

The energy source 119 could represent a source of energy that may, infull or in part, power the engine/motor 118. Examples of energy sources119 contemplated within the scope of the present disclosure includegasoline, diesel, other petroleum-based fuels, propane, other compressedgas-based fuels, ethanol, solar panels, batteries, and other sources ofelectrical power. The energy source(s) 119 could additionally oralternatively include any combination of fuel tanks, batteries,capacitors, and/or flywheels. The energy source 118 could also provideenergy for other systems of the vehicle 100.

The transmission 120 could include elements that are operable totransmit mechanical power from the engine/motor 118 to the wheels/tires121. The transmission 120 could include a gearbox, a clutch, adifferential, and a drive shaft. Other components of transmission 120are possible. The drive shafts could include one or more axles thatcould be coupled to the one or more wheels/tires 121.

The wheels/tires 121 of vehicle 100 could be configured in variousformats, including a unicycle, bicycle/motorcycle, tricycle, orcar/truck four-wheel format. Other wheel/tire geometries are possible,such as those including six or more wheels. Any combination of thewheels/tires 121 of vehicle 100 may be operable to rotate differentiallywith respect to other wheels/tires 121. The wheels/tires 121 couldrepresent at least one wheel that is fixedly attached to thetransmission 120 and at least one tire coupled to a rim of the wheelthat could make contact with the driving surface. The wheels/tires 121could include any combination of metal and rubber. Other materials arepossible.

The sensor system 104 may include several elements such as a GlobalPositioning System (GPS) 122, an inertial measurement unit (IMU) 124, aradar 126, a laser rangefinder/LIDAR 128, a camera 130, a steeringsensor 123, and a throttle/brake sensor 125. The sensor system 104 couldalso include other sensors, such as those that may monitor internalsystems of the vehicle 100 (e.g., O₂ monitor, fuel gauge, engine oiltemperature, brake wear).

The GPS 122 could include a transceiver operable to provide informationregarding the position of the vehicle 100 with respect to the Earth. TheIMU 124 could include a combination of accelerometers and gyroscopes andcould represent any number of systems that sense position andorientation changes of a body based on inertial acceleration.Additionally, the IMU 124 may be able to detect a pitch and yaw of thevehicle 100. The pitch and yaw may be detected while the vehicle isstationary or in motion.

The radar 126 may represent a system that utilizes radio signals tosense objects, and in some cases their speed and heading, within thelocal environment of the vehicle 100. Additionally, the radar 126 mayhave a plurality of antennas configured to transmit and receive radiosignals. The laser rangefinder/LIDAR 128 could include one or more lasersources, a laser scanner, and one or more detectors, among other systemcomponents. The laser rangefinder/LIDAR 128 could be configured tooperate in a coherent mode (e.g., using heterodyne detection) or in anincoherent detection mode. The camera 130 could include one or moredevices configured to capture a plurality of images of the environmentof the vehicle 100. The camera 130 could be a still camera or a videocamera.

The steering sensor 123 may represent a system that senses the steeringangle of the vehicle 100. In some embodiments, the steering sensor 123may measure the angle of the steering wheel itself. In otherembodiments, the steering sensor 123 may measure an electrical signalrepresentative of the angle of the steering wheel. Still, in furtherembodiments, the steering sensor 123 may measure an angle of the wheelsof the vehicle 100. For instance, an angle of the wheels with respect toa forward axis of the vehicle 100 could be sensed. Additionally, in yetfurther embodiments, the steering sensor 123 may measure a combination(or a subset) of the angle of the steering wheel, electrical signalrepresenting the angle of the steering wheel, and the angle of thewheels of vehicle 100.

The throttle/brake sensor 125 may represent a system that senses theposition of either the throttle position or brake position of thevehicle 100. In some embodiments, separate sensors may measure thethrottle position and brake position. In some embodiments, thethrottle/brake sensor 125 may measure the angle of both the gas pedal(throttle) and brake pedal. In other embodiments, the throttle/brakesensor 125 may measure an electrical signal that could represent, forinstance, an angle of a gas pedal (throttle) and/or an angle of a brakepedal. Still, in further embodiments, the throttle/brake sensor 125 maymeasure an angle of a throttle body of the vehicle 100. The throttlebody may include part of the physical mechanism that provides modulationof the energy source 119 to the engine/motor 118 (e.g., a butterflyvalve or carburetor). Additionally, the throttle/brake sensor 125 maymeasure a pressure of one or more brake pads on a rotor of vehicle 100.In yet further embodiments, the throttle/brake sensor 125 may measure acombination (or a subset) of the angle of the gas pedal (throttle) andbrake pedal, electrical signal representing the angle of the gas pedal(throttle) and brake pedal, the angle of the throttle body, and thepressure that at least one brake pad is applying to a rotor of vehicle100. In other embodiments, the throttle/brake sensor 125 could beconfigured to measure a pressure applied to a pedal of the vehicle, suchas a throttle or brake pedal.

The control system 106 could include various elements include steeringunit 132, throttle 134, brake unit 136, a sensor fusion algorithm 138, acomputer vision system 140, a navigation/pathing system 142, and anobstacle avoidance system 144. The steering unit 132 could represent anycombination of mechanisms that may be operable to adjust the heading ofvehicle 100. The throttle 134 could control, for instance, the operatingspeed of the engine/motor 118 and thus control the speed of the vehicle100. The brake unit 136 could be operable to decelerate the vehicle 100.The brake unit 136 could use friction to slow the wheels/tires 121. Inother embodiments, the brake unit 136 could convert the kinetic energyof the wheels/tires 121 to electric current.

A sensor fusion algorithm 138 could include, for instance, a Kalmanfilter, Bayesian network, or other algorithm that may accept data fromsensor system 104 as input. The sensor fusion algorithm 138 couldprovide various assessments based on the sensor data. Depending upon theembodiment, the assessments could include evaluations of individualobjects and/or features, evaluation of a particular situation, and/orevaluate possible impacts based on the particular situation. Otherassessments are possible.

The computer vision system 140 could include hardware and softwareoperable to process and analyze images in an effort to determineobjects, important environmental objects (e.g., stop lights, road wayboundaries, etc.), and obstacles. The computer vision system 140 coulduse object recognition, Structure From Motion (SFM), video tracking, andother algorithms used in computer vision, for instance, to recognizeobjects, map an environment, track objects, estimate the speed ofobjects, etc.

The navigation/pathing system 142 could be configured to determine adriving path for the vehicle 100. The navigation/pathing system 142 mayadditionally update the driving path dynamically while the vehicle 100is in operation. In some embodiments, the navigation/pathing system 142could incorporate data from the sensor fusion algorithm 138, the GPS122, and known maps so as to determine the driving path for vehicle 100.

The obstacle avoidance system 144 could represent a control systemconfigured to evaluate potential obstacles based on sensor data andcontrol the vehicle 100 to avoid or otherwise negotiate the potentialobstacles.

Various peripherals 108 could be included in vehicle 100. For example,peripherals 108 could include a wireless communication system 146, atouchscreen 148, a microphone 150, and/or a speaker 152. The peripherals108 could provide, for instance, means for a user of the vehicle 100 tointeract with the user interface 116. For example, the touchscreen 148could provide information to a user of vehicle 100. The user interface116 could also be operable to accept input from the user via thetouchscreen 148. In other instances, the peripherals 108 may providemeans for the vehicle 100 to communicate with devices within itsenvironment.

In one example, the wireless communication system 146 could beconfigured to wirelessly communicate with one or more devices directlyor via a communication network. For example, wireless communicationsystem 146 could use 3G cellular communication, such as CDMA, EVDO,GSM/GPRS, or 4G cellular communication, such as WiMAX or LTE.Alternatively, wireless communication system 146 could communicate witha wireless local area network (WLAN), for example, using WiFi. In someembodiments, wireless communication system 146 could communicatedirectly with a device, for example, using an infrared link, Bluetooth,or ZigBee. Other wireless protocols, such as various vehicularcommunication systems, are possible within the context of thedisclosure. For example, the wireless communication system 146 couldinclude one or more dedicated short-range communications (DSRC) devicesthat could include public and/or private data communications betweenvehicles and/or roadside stations.

The power supply 110 may provide power to various components of vehicle100 and could represent, for example, a rechargeable lithium-ion orlead-acid battery. In an example embodiment, one or more banks of suchbatteries could be configured to provide electrical power. Other powersupply materials and types are possible. Depending upon the embodiment,the power supply 110, and energy source 119 could be integrated into asingle energy source, such as in some all-electric cars.

Many or all of the functions of vehicle 100 could be controlled bycomputer system 112. Computer system 112 may include at least oneprocessor 113 (which could include at least one microprocessor) thatexecutes instructions 115 stored in a non-transitory computer readablemedium, such as the data storage 114. The computer system 112 may alsorepresent a plurality of computing devices that may serve to controlindividual components or subsystems of the vehicle 100 in a distributedfashion.

In some embodiments, data storage 114 may contain instructions 115(e.g., program logic) executable by the processor 113 to execute variousfunctions of vehicle 100, including those described above in connectionwith FIG. 1. Data storage 114 may contain additional instructions aswell, including instructions to transmit data to, receive data from,interact with, and/or control one or more of the propulsion system 102,the sensor system 104, the control system 106, and the peripherals 108.

In addition to the instructions 115, the data storage 114 may store datasuch as roadway maps, path information, among other information. Suchinformation may be used by vehicle 100 and computer system 112 duringthe operation of the vehicle 100 in the autonomous, semi-autonomous,and/or manual modes.

The vehicle 100 may include a user interface 116 for providinginformation to or receiving input from a user of vehicle 100. The userinterface 116 could control or enable control of content and/or thelayout of interactive images that could be displayed on the touchscreen148. Further, the user interface 116 could include one or moreinput/output devices within the set of peripherals 108, such as thewireless communication system 146, the touchscreen 148, the microphone150, and the speaker 152.

The computer system 112 may control the function of the vehicle 100based on inputs received from various subsystems (e.g., propulsionsystem 102, sensor system 104, and control system 106), as well as fromthe user interface 116. For example, the computer system 112 may utilizeinput from the sensor system 104 in order to estimate the outputproduced by the propulsion system 102 and the control system 106.Depending upon the embodiment, the computer system 112 could be operableto monitor many aspects of the vehicle 100 and its subsystems. In someembodiments, the computer system 112 may disable some or all functionsof the vehicle 100 based on signals received from sensor system 104.

The components of vehicle 100 could be configured to work in aninterconnected fashion with other components within or outside theirrespective systems. For instance, in an example embodiment, the camera130 could capture a plurality of images that could represent informationabout a state of an environment of the vehicle 100 operating in anautonomous mode. The state of the environment could include parametersof the road on which the vehicle is operating. For example, the computervision system 140 may be able to recognize the slope (grade) or otherfeatures based on the plurality of images of a roadway. Additionally,the combination of Global Positioning System 122 and the featuresrecognized by the computer vision system 140 may be used with map datastored in the data storage 114 to determine specific road parameters.Further, the radar unit 126 may also provide information about thesurroundings of the vehicle.

In other words, a combination of various sensors (which could be termedinput-indication and output-indication sensors) and the computer system112 could interact to provide an indication of an input provided tocontrol a vehicle or an indication of the surroundings of a vehicle.

In some embodiments, the computer system 112 may make a determinationabout various objects based on data that is provided by systems otherthan the radio system. For example, the vehicle may have lasers or otheroptical sensors configured to sense objects in a field of view of thevehicle. The computer system 112 may use the outputs from the varioussensors to determine information about objects in a field of view of thevehicle. The computer system 112 may determine distance and directioninformation to the various objects. The computer system 112 may alsodetermine whether objects are desirable or undesirable based on theoutputs from the various sensors.

Although FIG. 1 shows various components of vehicle 100, i.e., wirelesscommunication system 146, computer system 112, data storage 114, anduser interface 116, as being integrated into the vehicle 100, one ormore of these components could be mounted or associated separately fromthe vehicle 100. For example, data storage 114 could, in part or infull, exist separate from the vehicle 100. Thus, the vehicle 100 couldbe provided in the form of device elements that may be locatedseparately or together. The device elements that make up vehicle 100could be communicatively coupled together in a wired and/or wirelessfashion.

FIG. 2 shows a vehicle 200 that could be similar or identical to vehicle100 described in reference to FIG. 1. Depending on the embodiment,vehicle 200 could include a sensor unit 202, a wireless communicationsystem 204, a radio unit 206, a laser rangefinder 208, and a camera 210.The elements of vehicle 200 could include some or all of the elementsdescribed for FIG. 1. Although vehicle 200 is illustrated in FIG. 2 as acar, other embodiments are possible. For instance, the vehicle 200 couldrepresent a truck, a van, a semi-trailer truck, a motorcycle, a golfcart, an off-road vehicle, or a farm vehicle, among other examples.

The sensor unit 202 could include one or more different sensorsconfigured to capture information about an environment of the vehicle200. For example, sensor unit 202 could include any combination ofcameras, radars, LIDARs, range finders, radio devices (e.g., Bluetoothand/or 802.11), and acoustic sensors. Other types of sensors arepossible. Depending on the embodiment, the sensor unit 202 could includeone or more movable mounts that could be operable to adjust theorientation of one or more sensors in the sensor unit 202. In oneembodiment, the movable mount could include a rotating platform thatcould scan sensors so as to obtain information from each directionaround the vehicle 200. In another embodiment, the movable mount of thesensor unit 202 could be moveable in a scanning fashion within aparticular range of angles and/or azimuths. The sensor unit 202 could bemounted atop the roof of a car, for instance, however other mountinglocations are possible. Additionally, the sensors of sensor unit 202could be distributed in different locations and need not be collocatedin a single location. Some possible sensor types and mounting locationsinclude radio unit 206 and laser range finder 208.

The wireless communication system 204 could be located as depicted inFIG. 2. Alternatively, the wireless communication system 204 could belocated, fully or in part, elsewhere. The wireless communication system204 may include wireless transmitters and receivers that could beconfigured to communicate with devices external or internal to thevehicle 200. Specifically, the wireless communication system 204 couldinclude transceivers configured to communicate with other vehiclesand/or computing devices, for instance, in a vehicular communicationsystem or a roadway station. Examples of such vehicular communicationsystems include dedicated short-range communications (DSRC), radiofrequency identification (RFID), and other proposed communicationstandards directed towards intelligent transport systems.

The camera 210 could be mounted inside a front windshield of the vehicle200. The camera 210 could be configured to capture a plurality of imagesof the environment of the vehicle 200. Specifically, as illustrated, thecamera 210 could capture images from a forward-looking view with respectto the vehicle 200. Other mounting locations and viewing angles ofcamera 210 are possible. The camera 210 could represent one or morevisible light cameras. Alternatively or additionally, camera 210 couldinclude infrared sensing capabilities. The camera 210 could haveassociated optics that could be operable to provide an adjustable fieldof view. Further, the camera 210 could be mounted to vehicle 200 with amovable mount that could be operable to vary a pointing angle of thecamera 210.

A method 300 is provided for enabling a system to take advantage ofhuman operators (or a more powerful computer) as part of the objectidentification of the autonomous vehicle. The method could be performedusing any of the apparatus shown in FIGS. 1-2 and FIGS. 4-5 anddescribed herein; however, other configurations could be used as well.FIG. 3 illustrates the blocks in an example method. However, it isunderstood that in other embodiments, the blocks may appear in differentorder and blocks could be added, subtracted, or modified. Additionally,the blocks may be performed in a linear manner (as shown) or may beperformed in a parallel manner (not shown).

Block 302 includes the vehicle receiving data representing objects anenvironment in which the vehicle operates. In various embodiments, thevehicle may receive data representing objects in an environment in whichthe vehicle operates in a variety of ways. A sensor system on theautonomous vehicle may provide the data representing objects of theenvironment. For example, a vehicle may have various sensors, includinga camera, a radar unit, a laser range finder, a microphone, a radiounit, and other sensors. Each of these sensors may communicate data to aprocessor in the vehicle about information each respective sensorreceives.

In one example, a camera may be configured to capture still imagesand/or video. In various embodiments, the vehicle may have more than onecamera positioned in different orientations. Also, in some embodimentsthe camera may be able to move to capture images and/or video indifferent directions. The camera may be configured to store capturedimages and video to a memory for later processing by a processing systemof the vehicle. The captured images and/or video may be the dataindicating an environment.

In another example, a radar unit may be configured to transmit anelectromagnetic signal that will be reflected by various objects nearthe vehicle. The radar unit may be able to capture reflectedelectromagnetic signals. The captured reflected electromagnetic signalsmay enable the radar system (or processing system) to make variousdeterminations about objects that reflected the electromagnetic signal.For example, the distance and position to various reflecting objects maybe determined. In various embodiments, the vehicle may have more thanone camera in different orientations. The radar system may be configuredto store captured information to a memory for later processing by aprocessing system of the vehicle. The information captured by the radarsystem may be data indicating an environment.

In a yet further example, a laser range finder may be configured totransmit an electromagnetic signal (e.g., light, such as that from a gasor diode laser, or other possible light source) that will be reflectedby a target objects near the vehicle. The laser range finder may be ableto capture the reflected electromagnetic (e.g., laser) signals. Thecaptured reflected electromagnetic signals may enable the range-findingsystem (or processing system) to determine a range to various objects.The range-finding system may also be able to determine a velocity orspeed of target objects and store it as data indicating an environment.

Additionally, in an example, a microphone may be configured to captureaudio of environment surrounding the vehicle. Sounds captured by themicrophone may include emergency vehicle sirens and the sounds of othervehicles. For example, the microphone may capture the sound of the sirenof an emergency vehicle. A processing system may be able to identifythat the captured audio signal is indicative of an emergency vehicle. Inanother example, the microphone may capture the sound of an exhaust ofanother vehicle, such as that from a motorcycle. A processing system maybe able to identify that the captured audio signal is indicative of amotorcycle. The data captured by the microphone may form a portion ofthe data indicating an environment.

One more example has the radio unit being configured to transmit anelectromagnetic signal that may take the form of a Bluetooth signal,802.11 signal, and/or other radio technology signal. The firstelectromagnetic radiation signal may be transmitted via one or moreantennas located in a radio unit. Further, the first electromagneticradiation signal may be transmitted with one of many differentradio-signaling modes. However, in some embodiments it is desirable totransmit the first electromagnetic radiation signal with a signalingmode that requests a response from devices located near the autonomousvehicle. The processing system may be able to detect nearby devicesbased on the responses communicated back to the radio unit and use thiscommunicated information as a portion of the data indicating anenvironment.

In some embodiments, the processing system may be able to combineinformation from the various sensors in order to make furtherdeterminations of the environment of the vehicle. For example, theprocessing system may combine data from both radar information and acaptured image to determine if another vehicle or pedestrian is in frontof the autonomous vehicle. In other embodiments, other combinations ofsensor data may be used by the processing system to make determinationsabout the environment.

Block 304 includes a control system controlling the operation of thevehicle, while the vehicle operates in an autonomous mode. In someinstances, block 304 may be performed while block 302 is beingperformed. While operating in the autonomous mode, the vehicle may use acomputer system to control the operation of the vehicle withlittle-to-no human input. For example, a human-operator may enter anaddress into an autonomous vehicle and the vehicle may then be able todrive, without further input from the human (e.g., the human does nothave to steer or touch the brake/gas pedals), to the specifieddestination.

While the vehicle is operating autonomously, the sensor system may bereceiving data about the environment of the vehicle, as disclosed withrespect to block 302. The processing system of the vehicle may alter thecontrol of the vehicle based on data received from the various sensors.In some examples, the autonomous vehicle may alter a velocity of theautonomous vehicle in response to data from the various sensors. Theautonomous vehicle may change velocity in order to avoid obstacles, obeytraffic laws, etc. When a processing system in the vehicle identifiesobjects near the autonomous vehicle, the vehicle may be able to changevelocity, or alter the movement in another way.

As previously stated, there is often a trade-off that can be madebetween the precision of the system (how often it reports it sees anobject when it fact the object doesn't exist in reality) and the recallof the system (how often it misses an object when the object does existin reality). The system of the present disclosure includes a system biasto have high recall (at the expense of lower precision) so that thesystem will generally default toward indicating the presence of anobject even when the system has low confidence. When the system detectsan object but is not highly confident in the detection of the object,the system can ask a human operator (or a more powerful computer) toconfirm that the object does in fact exist. Thus, the system willgenerally operate as a highly accurate autonomous system. For example,in some embodiments, even when the vehicle has a low confidence of adetected object, it may alter a movement of the autonomous vehicle inorder to make sure the vehicle is operated safely.

Block 306 includes the vehicle analyzing the data representing objectsof the environment to determine at least one object having a detectionconfidence below a threshold. A processor in the vehicle may beconfigured to calculate various objects of the environment based on datafrom the various sensors. For example, in one embodiment, the processoris configured to detect objects that may be important for an autonomousvehicle to recognize. Objects may include pedestrians, street signs,other vehicles, indicator signals on other vehicles, and other variousobjects detected in the captured data.

The detection confidence may indicate or be indicative of a likelihoodthat the determined object is correctly identified in the environment,or is present in the environment. For example, the processor may performobject detection of objects within image data in the received data, anddetermine that the at least one object has the detection confidencebelow the threshold based on being unable to identify the object with adetection confidence above the threshold. If a result of an objectdetection or object recognition of the object is inconclusive, then thedetection confidence may be low or below the set threshold.

The vehicle may detect objects of the environment in various waydepending on the source of the data. In some embodiments, the datarepresenting objects of the environment may come from a camera and beimage (or video) data. In other embodiments, the data representingobjects of the environment may come from a LIDAR unit. The vehicle mayanalyze the captured image or video data to identify objects. Thevehicle may include methods and apparatuses configured to detect objectsin image or video data. The methods and apparatuses may be configured tomonitor image and/or video data for the presence of objects of theenvironment. In other embodiments, the data representing objects of theenvironment may be radar, audio, or other data. The vehicle may beconfigured to identify objects of the environment based on the radar,audio, or other data. In these embodiments, vehicle may include methodsand apparatuses configured to detect objects based on the type of data.In some embodiments, the detection methods the vehicle uses may be basedon a set of known data. For example, data related to environmentalobjects may be stored to the vehicle. The vehicle may compare receiveddata to the stored data to determine objects. In other embodiments, thevehicle may be configured to determine objects based on the context ofthe data. For example, street signs related to construction maygenerally have an orange color. The vehicle may be configured to detectobjects that are orange, and located near the side of roadways asconstruction-related street signs. Additionally, when the processingsystem of the autonomous vehicle detects objects in the captured data,it also may calculate a confidence for each object.

Further, the autonomous vehicle may also have a confidence threshold.The confidence threshold may vary depending on the type of object beingdetected. For example, the confidence threshold may be lower for someobject that may require a quick action from the autonomous vehicle, suchas brake lights on another vehicle. However, in other embodiments, theconfidence threshold may be the same for all detected objects. If theconfidence associated with a detected object is greater than theconfidence threshold, then the vehicle may assume the future wascorrectly recognized and adjust the control of the autonomous vehicle tocompensate. How the vehicle reacts when the confidence associated with adetected object is less than the confidence threshold depends on thespecific embodiment. In some embodiments, the vehicle may react as ifthey detected object exists despite the low confidence level. In otherembodiments, the vehicle may react as if the future was detectedincorrectly, and does not really present.

When the vehicle detects an object of the environment, it may alsocalculate a confidence associated with the specific detected object. Theconfidence may be calculated in various ways depending on theembodiment. In one example, when detecting objects of the environment,the vehicle may compare data representing objects of the environment topredetermined data relating to known objects. The closer the matchbetween the data representing objects of the environment topredetermined data, the higher the confidence. In other embodiments, thevehicle may detect objects based on a mathematical analysis of the datarepresenting objects of the environment. The mathematical analysis maycreate a confidence based on the mathematical analysis.

Block 308 includes the vehicle communicating at least a subset of thedata indicating the environment for further processing to asecondary-processing device. Because detecting objects of theenvironment can be important to the operation of an autonomous vehicle,an object with a confidence below the threshold may have the dataassociated with it communicated for further processing. In someembodiments, the subset of the data communicated may be image datacaptured by the vehicle. Thus, the image data of the object can becommunicated to a remote operator for further processing. In someexamples, a bounding box can be providing substantially around theobject in the image data, and then the image data is communicated to theremote operator so that the remote operator can readily and quicklyidentify the object in the image data (e.g., the object in the boundingbox) for which there is a question.

In some embodiments, when the object is detected as having a confidencebelow the confidence threshold, the object may be given a preliminaryidentification. The control system of the vehicle may adjust theoperation of the vehicle in response to this preliminary identification.For example, altering the movement of the vehicle may include stoppingthe vehicle, switching the vehicle to a human-controlled mode, changinga velocity of vehicle (e.g., a speed and/or direction), or othermovement alteration.

In additional embodiments, when the object is detected as having aconfidence below the confidence threshold, a processing device locatedin either the vehicle or in another computing system may generate anatural-language question based on the identification of the object. Forexample, the natural-language question may be, “Is this a stop sign?” Inother examples, the natural-language question may take other forms suchas, “Is this a construction sign?” Other various natural-languagequestions may be generated based on the detected object. The naturallanguage question may be based on a result of the object detection ofthe object, or based on the preliminary identification of the object, soas to ask the remote operator to confirm or deny whether the preliminaryidentification is correct.

Additionally, the natural-language question may be communicated forfurther processing to the secondary-processing device.

In various embodiments, the further processing with thesecondary-processing device may take different forms. In one embodiment,a human operator provides input to the secondary-processing device aspart of the further processing. The human operator can either confirmthe identified object with the low confidence or the human operator cancorrectly identify the object. The human operator may be located withinthe autonomous vehicle, such as a passenger in the passenger seat or aperson sitting in the driver's seat. Or, in other embodiments, the humanoperate may be located in a location that is not near the autonomousvehicle. For example, a remote human operator may be located at a remotecomputer terminal. This human operator may receive data from one or moreautonomous vehicle and provide an identification of various objects foreach autonomous vehicle that communicates data to the human operator.

The human operator may be provided with a representation of the subsetof the data indicating the environment for further processing (e.g., animage). The human operator may provide an input indicating a correctidentification by the autonomous vehicle processing system, or mayindicate the system identified the object incorrectly. If the systemidentified the object incorrectly, the human operator may be able toprovide a correct identification. For example, the human operator may bepresented with both the natural-language question and captured imagedata. The human operator may answer the natural-language question basedon the human operator inspecting the image. For example, the humanoperator may indicate yes or no based on his or her perception of animage when the natural-language question asks, “Is this a stop sign?”

In other embodiments, a more powerful computer, such as a remotecomputer server, performs the further processing. The more powerfulcomputer can either confirm the identified object with the lowconfidence or the more powerful computer can correctly identify theobject. The more powerful computer may have a much larger computationalpower than the computing system in the autonomous vehicle, thus the morepowerful computer may able to more accurately identify objects of theenvironment. In some embodiments the autonomous vehicle may wirelesslycommunicate the subset of the data indicating the environment to theremote computing device. The more powerful computer may confirm acorrect identification by the autonomous vehicle processing system, orthe more powerful computer may indicate the system identified the objectincorrectly. If the system identified the object incorrectly, the morepowerful computer may be able to provide a correct identification.

In further embodiments, the more powerful computer may also not be ableto correctly identify the object of the environment. When this happens,the system may fall back to a human operator performs the furtherprocessing. In this embodiment, the identification may have threestages. First, the computing system in the autonomous vehicle mayattempt to identify the object. If the identification is below athreshold confidence, a more powerful computer may be used in attempt toidentify the object of the environment. If the more powerful computeralso classifies the object below the threshold confidence, a humanoperator performs the further processing to provide a identification forthe object of the environment. Therefore, the human operator and themore powerful computer may work in tandem to identify objects of theenvironment of the vehicle.

Additionally, in some embodiments, the more powerful computer and thehuman operator may both receive data when the identified object has lowconfidence. Therefore, either the more powerful computer or the humanoperator (and possibly both) may be able to either confirm theidentified object with the low confidence or correctly identify theobject. By transmitting data to both the more powerful computer and thehuman operator at approximately the same time, any latency issues witheither the more powerful computer or the human operator may beminimized. Further, when an object has been identified, a global mapthat all vehicles may access may be updated to include the newidentified object. In some other embodiments, vehicles may communicatethe identification of the object to other nearby vehicles.

Block 310 includes the vehicle receiving an indication of an objectconfirmation of the subset of the data. The vehicle may receive anindication of an object confirmation of the subset of the data in avariety of ways. Because either a human operator or a more powerfulcomputer may determine the indication of an object confirmation of thesubset of the data, the source of the indication of an objectconfirmation of the subset of the data may change. Therefore, thereceiving may be different in various embodiments. An objectconfirmation includes an indication of an object that is present in theat least a subset of the data that was communicated further processingto a secondary-processing device at Block 308. For example, the objectconfirmation may be a correct identification of road sign. In anothersituation, the object confirmation may include an identification that anaudio sample contains an emergency vehicle siren signal. In someexamples, the remote operator may provide an answer to thenatural-language question providing an object indication (e.g.,indicating that the object is in fact a stop sign).

In some embodiments, such as those where the further processing of block308 is performed at a location collocated with the vehicle; thereceiving an object confirmation includes an occupant of the vehicleentering information into a computer system. Thus, in some embodimentsthe receiving may be performed when a person located in the vehicleprovides an input or information.

In other embodiments, such as those where the further processing ofblock 308 is performed at a location that is not collocated with thevehicle, the receiving an object confirmation may include wirelesslyreceiving the object confirmation. In some embodiments, even when thefurther processing of block 308 is performed at a location that iscollocated with the vehicle, the receiving an object confirmation of thesubset of the data may include wirelessly receiving the objectconfirmation. For example, the object confirmation may be transmitted tothe autonomous vehicle from a remote more powerful computer system orfrom a remote human operator. In this example, because the more powerfulcomputer system or from a remote human operator is not collocated withthe autonomous vehicle. Therefore, the object confirmation may bewirelessly communicated to the autonomous vehicle.

In another example, a human operator who is collocated with theautonomous vehicle, such as a passenger, may possess a mobile device.The human operator may cause the object confirmation to be transmittedto the autonomous vehicle from the mobile device. Therefore, thereceiving an object confirmation of the subset of the data may includewirelessly receiving the indication even when the further processing ofblock 308 is performed at a location that is collocated with thevehicle.

Block 312 includes the vehicle altering the control of the apparatus bythe control system based on the object confirmation of the subset of thedata. When the vehicle receives the object confirmation of the subset ofthe data, the object confirmation may be communicated to the controlsystem of the vehicle. Based on the object confirmation, the vehicle mayalter the control of the vehicle. The control of the vehicle may bealtered based on the object confirmation of the subset of data. In otherexamples, the processor may provide instructions to control theautonomous vehicle by the control system based on the object indicationreceived from the remote operator.

In some examples, the object confirmation of the subset of the data mayindicate the presence of an object that the autonomous vehicle was notaware of before it received the object confirmation of the subset of thedata. In another embodiment, the object confirmation of the subset ofthe data may indicate that an object is different from how theprocessing system identified the object. In yet a further embodiment,the object confirmation of the subset of the data may indicate an objectidentified by the autonomous vehicle was not actually present in theenvironment (e.g., a false positive). In each of these three examples,the object confirmation of the subset of the data provides informationto the autonomous vehicle that has different objects than the autonomousvehicle determined. Therefore, to continue safe operation of theautonomous vehicle, the control of the vehicle may be altered.

For example, altering the movement of the vehicle may include stoppingthe vehicle, switching the vehicle to a human-controlled mode, changinga velocity of vehicle (e.g., a speed and/or direction), or othermovement alteration.

FIG. 4A illustrates a top view of a scenario encountered by anautonomous vehicle, in accordance with an example embodiment. As shown,an autonomous vehicle 402 may be operating within an environment 400containing other vehicles 406, 408, and 410. The autonomous vehicle 402may be operating in an autonomous mode with a lane of travel when itapproaches an obstacle in the road, in this example a temporary stopsign 404.

The autonomous vehicle 402 may create a representation of itsenvironment 400 based on any combination of possible types of sensordata as described above. FIG. 4B illustrates a representation of theenvironment from FIG. 4A based on sensor data collected by the vehicle,according to an example embodiment. In some examples, the representationmay not be a perfect copy of the environment. For instance, some of thesensors may be blocked in certain directions or some of the sensor datamay be distorted. Additionally, some objects may be abstracted intogeometric shapes, such as the representations of the vehicles 406, 408,and 410 or the temporary stop sign 404 shown in the figure. Theautonomous vehicle 402 may identify objects or other aspects of theenvironment with varying levels of precision.

The situation depicted in FIG. 4A and FIG. 4B may be a situation inwhich the vehicle's confidence level drops below a predeterminedthreshold level. The drop in confidence may be based on one or moredifferent factors about the vehicle's operation and/or the vehicle'sview of the environment. For example, the vehicle 402 may not be able tocreate a complete sensor representation of its environment because thetemporary stop sign 404 may be obstructing its views of aspects of theenvironment (e.g., other cars). Additionally, the vehicle 402 may not beable to identify with confidence one or more objects within theenvironment, possibly including the temporary stop sign 404. Also,aspects of the vehicle's own operation may also cause its confidencelevel to drop. For instance, the vehicle may have stopped behind thetemporary stop sign 404, and may have remained stuck there for a certainperiod of time, which may trigger a warning from one of the vehicle'ssystems. In some examples, if the vehicle 402 is stuck for more than apredetermined set amount of time (e.g., 1 minute or 5 minutes), itsconfidence level may begin to drop. Other factors may contribute to thevehicle's determination that its confidence in how to proceed (e.g.,whether to continue waiting or to do something else) has fallen to alevel where the vehicle should request remote assistance.

FIG. 4C shows a video stream of the environment 400 of autonomousvehicle 402 from the point-of-view of the autonomous vehicle 402. Forexample, the autonomous vehicle 402 may be equipped with one or morevideo cameras which capture video streams of a portion of theenvironment 400. This data may be transmitted along with the requestwith assistance for use by the remote operator. In this example, theportion of the environment 400 captured in the video stream includes thetemporary stop sign 404 as well as parts of cars 408 and 410 that arenot obstructed by the temporary stop sign 404. In some examples, thecameras may be moveable (and possibly may be controlled directly orindirectly by a remote operator) in order to capture video of additionalportions of the environment 400 in order to resolve certain scenarios.

In further examples, the request for assistance may additionally includeone or more suggested autonomous operations for the vehicle to take inthe identified situation. For example, referring back to the scenariodescribed with respect to FIG. 4, the vehicle may transmit options thatmay include holding position or attempting to pass the obstacle on theleft. In one example, the vehicle may send a single suggested operationin order to receive verification of its proposed course of action, andmay hold position until a response is received. In other examples, thevehicle may send a set of two or more proposed options for the remoteassistor to select from. In some cases, the vehicle may not be able topropose a course of action. In such examples, the human guide may beable to propose a course of action for the vehicle to take, or a set oftwo or more possible courses of action.

In additional examples, the request for assistance may involve multipleparts. For example, the vehicle may ask a series of questions of theremote assistor in order to determine how to proceed with operation. Forexample, a user interface may include a natural-language question to aidin providing the input to the autonomous vehicle. For example, referringto the situation depicted in FIG. 4A, the vehicle 402 may first requestassistance in order to identify the obstacle in the road as a temporarystop sign 404. The vehicle 402 may then make a second request in orderto determine how best to proceed given that the obstacle has beenidentified as a temporary stop sign 404. Other more complicateddiscourses between the vehicle 402 and remote operator are possible aswell.

In some examples, the human operator may be located in a remote locationthat has a wireless connection with a communication system of thevehicle. For example, a remote human operator may be located at a remotecomputer terminal with a graphical interface that provides informationfrom the autonomous vehicle in order for the human operator to answerthe request. For instance, FIG. 4D shows one example graphical interfacethat may be presented to a human operator. The graphical interface 412may include separate sub-windows 414 and 416. The first sub-window 414may include the vehicle's sensor data representation of its environment,such as described above with respect to FIG. 4B. The second sub-window416 may include a video stream of a portion of the environment, such asdescribed above with respect to FIG. 4C. Accordingly, the human operatormay be able to compare the vehicle's understanding of its environmentwith the video stream to verify the vehicle's representation of itsenvironment and/or to verify or suggest a planned course of action ofthe vehicle.

The remote assistor may be presented with a graphical interface thatcontains a control menu that enables a remote assistor to send aresponse to a vehicle indicating a proposed autonomous mode ofoperation. For example, FIG. 4E shows an example graphical interfacethat contains a first sub-window showing the vehicle's sensor datarepresentation of its environment and a second sub-window showing avideo stream of a portion of the vehicle's environment, such asdescribed above with respect to FIG. 4D. FIG. 4E additionally contains acontrol menu 418 that may allow a human operator to respond to anatural-language question. Depending on the type of response provided tothe vehicle, the control menu 418 may allow the operator to inputguidance to the vehicle in a number of different ways (e.g., selectingfrom a list of operations, typing in a particular mode of operation,selecting a particular region of focus within an image of theenvironment, etc.).

In the example depicted in FIG. 4E, the human guide may indicate anatural-language question 420 to identify the object identified as thetemporary stop sign 404. Additionally, when an identification isconfirmed, the identification may be added to a global map. When theidentification is added to the global map, other vehicles may not haveto request an identification of the object in the future. The controlmenu 418 may additionally contain a latency bar 422 indicating how oldthe received sensor data is, which may affect the human guide'sresponse.

The response to the request for assistance may be received in a numberof different ways. In cases where the request for assistance was sent toa remote assistor (or a remote computing system) not located within thevehicle, the response may be received wirelessly through a communicationsystem located within the autonomous vehicle. In other embodiments, suchas those where the request for assistance was sent to a passengerlocated with the vehicle, the response may be received when thepassenger enters an autonomous operation into a graphical interface of acomputer system located within the vehicle. A passenger may be able toinstruct the vehicle in other ways as well, such as through voicecommands or through a handheld mobile device. Other modes oftransmitting and/or receiving the request for assistance and/or theresponse to the request may also be used.

FIG. 5 illustrates an example scenario 500 involving a vehicle 502traveling down a roadway 504. Vehicle 502 could be operating in anautonomous mode. Further, the vehicle 502 may be configured with asensor unit 510. In one example, the sensor unit 510 may have a sensor,such as a camera, that has a field of view 506A. The field of view 506Amay correspond to a region of where the camera may be able to capture animage. In another embodiment, sensor unit 510 may include a radar unit.The field of view 506A may correspond to a region over which the radarunit may send and receive signals. In other embodiments, the field ofview 506A may not be limited to a single region in front of the vehicle,but instead may correspond to the entire region (e.g., 360-degrees)around the vehicle. FIG. 5 illustrates an example scenario 500 in whichthe sensor unit uses a camera to obtain data about the environment ofthe vehicle. The description of FIG. 5 can also be used with othersensors, not just an optical sensor like a camera.

As one example embodiment, as shown FIG. 5, there may be twoenvironmental objects at least partially within the field of view 506Aof a vehicle 502. In this example, it is assumed that the field of view506A is that of an optical sensor, such as a camera. The camera of thesensor unit 510 may take a picture or video. This picture video will beanalyzed to determine objects of the environment.

When the camera in the sensor unit 510 captures a video or image, afirst object 514 may fall completely within the field of view 506A. Asecond object 512 may only partially be located within the capturepicture or video. When a processing system in the vehicle 502 analyzesthe picture or video, it may be able to successfully identify an object,such as the first object 514. However, the processing system may not beable to successfully identify the second object 512 (or it may identifythe object 512 with a low confidence). The processing system may not beable to successfully identify the second object 512 for many differentreasons. In some embodiments, the data of the environment may notinclude enough information to successfully identify the second object512 automatically. For example, the second object 512 may be a streetsign. An image captured by the vehicle may have a portion of the streetsign cut off. The detection system of the vehicle may not be able tocorrectly identify the cut off street sign. In another example, anobject may be partially obscured, so automatic identification may notwork accurately. In still another embodiment, an object may be deformedor damaged in such a way that the detection system of the vehicle maynot be able to accurately detect the object.

Thus, the processing system may communicate data associated with thecaptured image or video for further processing. When a human views theresulting image or video, he or she may be able to successfully identifythe second object 512, despite the future only partially being in thefield of view 506A. In other embodiments, rather than communicating datato a human, the vehicle may communicate data to a more powerful computersystem, which is remotely located, for further processing.

Although FIG. 5 was described with respect to pictures and video, thesensor unit 510 may have other sensors, which capture data that is notvisible light. Therefore, the disclosed methods and apparatuses are notlimited to just optical data collection. Additionally, theidentification shown in FIG. 5 was described as having amisidentification due to the second future 512 only partially beingwithin the field of view. In some embodiments, a misidentification canoccur even though the full object is located in an image or video. Itdoesn't need to only partially up here in the image or video.

It will be understood that there are other similar methods that coulddescribe receiving data representative of an electromagnetic signal,receiving an indication of a movement of the vehicle, determining amovement parameter based the indication of the movement of the vehicle,and recovering the distance and direction information from theelectromagnetic signal, based on the movement parameter. Those similarmethods are implicitly contemplated herein.

In some embodiments, the disclosed methods may be implemented ascomputer program instructions encoded on a non-transitorycomputer-readable storage media in a machine-readable format, or onother non-transitory media or articles of manufacture. FIG. 6 is aschematic illustrating a conceptual partial view of an example computerprogram product that includes a computer program for executing acomputer process on a computing device, arranged according to at leastsome embodiments presented herein.

In one embodiment, the example computer program product 600 is providedusing a signal bearing medium 602. The signal bearing medium 602 mayinclude one or more programming instructions 604 that, when executed byone or more processors may provide functionality or portions of thefunctionality described above with respect to FIGS. 1-5. In someexamples, the signal bearing medium 602 may encompass a non-transitorycomputer-readable medium 606, such as, but not limited to, a hard diskdrive, a Compact Disc (CD), a Digital Video Disk (DVD), a digital tape,memory, etc. In some implementations, the signal bearing medium 602 mayencompass a computer recordable medium 608, such as, but not limited to,memory, read/write (R/W) CDs, R/W DVDs, etc. In some implementations,the signal bearing medium 602 may encompass a communications medium 610,such as, but not limited to, a digital and/or an analog communicationmedium (e.g., a fiber optic cable, a waveguide, a wired communicationslink, a wireless communication link, etc.). Thus, for example, thesignal bearing medium 602 may be conveyed by a wireless form of thecommunications medium 610.

The one or more programming instructions 604 may be, for example,computer executable and/or logic implemented instructions. In someexamples, a computing device such as the computer system 112 of FIG. 1may be configured to provide various operations, functions, or actionsin response to the programming instructions 604 conveyed to the computersystem 112 by one or more of the computer readable medium 606, thecomputer recordable medium 608, and/or the communications medium 610.

The non-transitory computer readable medium could also be distributedamong multiple data storage elements, which could be remotely locatedfrom each other. The computing device that executes some or all of thestored instructions could be a vehicle, such as the vehicle 200illustrated in FIG. 2. Alternatively, the computing device that executessome or all of the stored instructions could be another computingdevice, such as a server.

The above detailed description describes various features and functionsof the disclosed systems, devices, and methods with reference to theaccompanying figures. While various aspects and embodiments have beendisclosed herein, other aspects and embodiments will be apparent. Thevarious aspects and embodiments disclosed herein are for purposes ofillustration and are not intended to be limiting, with the true scopebeing indicated by the following claims.

What is claimed is:
 1. An autonomous vehicle comprising: a set ofsensors configured to receive data, wherein the data represents objectsof an environment of the autonomous vehicle; a control system,configured to operate the autonomous vehicle; and one or more processorsconfigured to: determine, based on the received data, at least oneobject of the environment that has a detection confidence below athreshold, wherein the detection confidence is indicative of alikelihood that the determined object is correctly identified in theenvironment; and based on the at least one object having a detectionconfidence below the threshold: communicate, from the received data,image data of the at least one object for further processing to a remoteoperator, determine at least one natural-language question associatedwith the at least one object having a detection confidence below thethreshold, communicate the at least one natural-language question to theremote operator, receive an answer to the at least one natural-languagequestion from the remote operator providing an object indication, andprovide instructions to control the autonomous vehicle by the controlsystem based on the object indication.
 2. The apparatus according toclaim 1, wherein the one or more processors are further configured to:provide a bounding box substantially around the at least one object inthe image data; and communicate the image data of the least one objectincluding the bounding box substantially around the at least one objectfor further processing to the remote operator.
 3. The apparatusaccording to claim 1, wherein the one or more processors are furtherconfigured to: perform object detection of objects within image data inthe received data; and determine that the at least one object has thedetection confidence below the threshold based on being unable toidentify the at least one object with a detection confidence above thethreshold.
 4. The apparatus according to claim 3, wherein the one ormore processors are further configured to: determine the at least onenatural-language question associated with the at least one object havingthe detection confidence below the threshold based on a result of theobject detection of the at least one object, wherein the at least onenatural-language question indicates a possible identification of the atleast one object and requests confirmation.
 5. The apparatus accordingto claim 1, wherein the set of sensors comprises a LIDAR sensor and thedata indicating the environment comprises LIDAR data.
 6. The apparatusaccording to claim 1, wherein the set of sensors comprises a camera andthe data indicating the environment comprises an image.
 7. The apparatusaccording claim 1, wherein the set of sensors comprises a microphone andthe data indicating the environment comprises an audio signal.
 8. Theapparatus according claim 1, wherein providing the instructions tocontrol the autonomous vehicle by the control system based on the objectindication comprises providing instructions to switch a mode of theautonomous vehicle to a human-controlled mode.
 9. A method comprising:receiving data by a set of sensors, wherein the data represents objectsof an environment of an autonomous vehicle; operating the autonomousvehicle by a control system; determining, by a processor, based on thereceived data, at least one object of the environment that has adetection confidence below a threshold, wherein the detection confidenceis indicative of a likelihood that the determined object is correctlyidentified in the environment; and based on the at least one objecthaving a detection confidence below the threshold, the processor:communicating, from the received data, image data of the at least oneobject for further processing to a remote operator, determining at leastone natural-language question associated with the at least one objecthaving a detection confidence below the threshold, communicating the atleast one natural-language question to the remote operator, receiving ananswer to the at least one natural-language question from the remoteoperator providing an object indication, and providing instructions tocontrol the autonomous vehicle by the control system based on the objectindication.
 10. The method according to claim 9, further comprising:providing a bounding box substantially around the at least one object inthe image data; and communicating the image data of the least one objectincluding the bounding box substantially around the at least one objectfor further processing to the remote operator.
 11. The method accordingto claim 9, further comprising: performing object detection of objectswithin image data in the received data; and determining that the atleast one object has the detection confidence below the threshold basedon being unable to identify the at least one object with a detectionconfidence above the threshold.
 12. The method according to claim 11,further comprising: determining the at least one natural-languagequestion associated with the at least one object having the detectionconfidence below the threshold based on a result of the object detectionof the at least one object, wherein the at least one natural-languagequestion indicates a possible identification of the at least one objectand requests confirmation.
 13. The method according to claim 9, whereinthe set of sensors comprises a LIDAR sensor and the data indicating theenvironment comprises LIDAR data.
 14. The method according to claim 9,wherein the set of sensors comprises a camera and the data indicatingthe environment comprises an image.
 15. The method according claim 9,wherein the set of sensors comprises a microphone and the dataindicating the environment comprises an audio signal.
 16. The methodaccording claim 9, wherein providing the instructions to control theautonomous vehicle by the control system based on the object indicationcomprises providing instructions to switch a mode of the autonomousvehicle to a human-controlled mode.
 17. An article of manufactureincluding a non-transitory computer-readable medium having storedthereon instructions that, when executed by a processor in a vehiclesystem, causes the vehicle system to perform operations comprising:receiving data by a set of sensors, wherein the data represents objectsof an environment of an autonomous vehicle; determining based on thereceived data, at least one object of the environment that has adetection confidence below a threshold, wherein the detection confidenceis indicative of a likelihood that the determined object is correctlyidentified in the environment; and based on the at least one objecthaving a detection confidence below the threshold: communicating, fromthe received data, image data of the at least one object for furtherprocessing to a remote operator, determining at least onenatural-language question associated with the at least one object havinga detection confidence below the threshold, communicating the at leastone natural-language question to the remote operator, receiving ananswer to the at least one natural-language question from the remoteoperator providing an object indication, and providing instructions tocontrol the autonomous vehicle by the control system based on the objectindication.
 18. The article of manufacture according to claim 17,wherein the operations further comprise: providing a bounding boxsubstantially around the at least one object in the image data; andcommunicating the image data of the least one object including thebounding box substantially around the at least one object for furtherprocessing to the remote operator.
 19. The article of manufactureaccording to claim 17, wherein the operations further comprise:performing object detection of objects within image data in the receiveddata; and determining that the at least one object has the detectionconfidence below the threshold based on being unable to identify the atleast one object with a detection confidence above the threshold. 20.The article of manufacture according to claim 17, wherein the operationsfurther comprise: determining the at least one natural-language questionassociated with the at least one object having the detection confidencebelow the threshold based on a result of the object detection of the atleast one object, wherein the at least one natural-language questionindicates a possible identification of the at least one object andrequests confirmation.
 21. A method comprising: determining, by aprocessor, based on received data, a preliminary identification of atleast one object of the environment; operating an autonomous vehicle bya control system, based on the preliminary identification; anddetermining, by a processor the at least one object of the environmentthat has a detection confidence below a threshold, wherein the detectionconfidence indicates a likelihood that the determined object is presentin the environment; and based on the at least one object having adetection confidence below the threshold, the processor: communicating,from the received data, image data of the at least one object forfurther processing to a remote operator, determining at least onenatural-language question associated with the at least one object havinga detection confidence below the threshold, communicating the at leastone natural-language question to the remote operator, receiving ananswer to the at least one natural-language question from the remoteoperator providing a corrected identification, and providinginstructions to control the autonomous vehicle by the control systembased on the corrected identification.
 22. The method according to claim21, further comprising: performing object detection of objects withinimage data in the received data; and determining that the at least oneobject has the detection confidence below the threshold based on beingunable to identify the at least one object with a detection confidenceabove the threshold.
 23. The method according to claim 21, furthercomprising: determining the at least one natural-language questionassociated with the at least one object having the detection confidencebelow the threshold based on a result of the object detection of the atleast one object, wherein the at least one natural-language questionindicates a possible identification of the at least one object andrequests confirmation.
 24. The method according to claim 21, whereinaltering the instructions to control of the autonomous vehicle comprisesswitching the autonomous vehicle to a human-controlled mode.